\documentclass[a4paper,11pt]{article}

\usepackage[pdftex]{graphicx}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{times}
\usepackage{amsmath}
\usepackage[hyphens]{url}
\urlstyle{same}
\usepackage{enumerate}
\usepackage{parskip}
\usepackage[colorlinks,urlcolor=navyblue]{hyperref}
\usepackage{microtype}
\usepackage{enumitem}% http://ctan.org/pkg/enumitem

\usepackage{color}
\definecolor{navyblue}{rgb}{0,0,0.5}

% if not draft, smaller printable area makes the paper more readable
\topmargin -4mm
\oddsidemargin 0mm
\textheight 225mm
\textwidth 150mm

%\parskip=\baselineskip

\pagestyle{empty}

\begin{document}
\thispagestyle{empty}

\section*{Instructions -- project work}
%\begin{itemize}[noitemsep,topsep=0pt]
Choose a data set and make all the steps of Bayesian data
  analysis workflow as explained below. 
  
\subsection*{Project notebook}
  
  The Project outcome is a R or Python notebook similar to notebooks in
\begin{itemize}[noitemsep,topsep=0pt]
  \item BDA R demos \url{https://github.com/avehtari/BDA_R_demos/tree/master/demos_rstan}
  \item BDA Python demos \url{https://github.com/avehtari/BDA_py_demos/tree/master/demos_pystan}
  \item Stan case studies \url{http://mc-stan.org/users/documentation/case-studies.html}
  \item StanCon case studies \url{http://mc-stan.org/users/documentation/case-studies.html}
  (some of these notebooks are for a bigger projects, but reflect still the basic idea of a notebook presentation)
\end{itemize}

The submitted notebooks need to illustrate the knowledge of the
  Bayesian workflow. It has to include:
\begin{itemize}
  \item Description of the data, and the analysis problem
  \item Description of at least two models, for example:
    \begin{itemize}
    \item non-hierarchical and hierarchical
    \item linear and non-linear
    \item variable selection with many models
    \end{itemize}
  \item Informative or weakly informative priors, and description of the prior choices
  \item Stan code
  \item How Stan model is run
  \item Convergence diagnostics (Rhat, divergences, ESS)
  \item Posterior predictive checking
  \item Model comparison (e.g. with loo)
  \item Predictive performance assessment if applicable
    (e.g. classification accuracy)
  \item Sensitivity analysis with respect to prior choices
  \item Discussion of problems, and potential improvements 
\end{itemize}

Group composition:
\begin{itemize}
  \item We generally recommend 2 person groups
  \item 1 or 3 person groups are also fine if not otherwise possible or sensible
  \item 3 person groups are expected to choose more difficult projects
  \item 2-3 person groups are highly recommended over 1 person groups. 2-3 person groups have priority when reserving presentation slots.
\end{itemize}

\subsection*{Project Presentation}

Presentation details:
\begin{itemize}
  \item Each project needs to be presented in addition to submitting the notebook
  \item The presentation should be high level but sufficiently detailed information should be readily available to facilitate answering questions from the audience
  \item Within each session, about four groups will be presenting
  \item For 1-2 person groups, the presentation should be 10 minutes
  \item For 3 person groups, the presentation should be 15 minutes 
  \item Afterwards, questions will be asked first by other students and then by two attending TAs for about 5 to 10 minutes
  \item Grading of the presentation will be done by the two TAs using standardized grading instructions
  \item Presenters' ID cards will be checked to ensure the right persons are presenting
\end{itemize}

Specific presentation recommendations:
\begin{itemize}
  \item The first slide needs to include project title and names of the group members.
  \item Big enough font size for text and figure labels should be used so that slides are easily readable for the audience.
  \item The last slide needs to include to conclusion and names of the group members. 
  \item If the last slide has only "Thanks", "Thank you", "Questions?" or something similar of zerocontent, the project grade is reduced by one. So we recommend you avoid such slides (in general).
  \item The best presentations we have seen were by groups who did visit TA session(s) and showed intermediate results to TAs and did get feedback how to improve. So we recommend you visiting TA sessions for the project.
\end{itemize}

\subsection*{Data sets and modelling recommendations}

Suggestions for sensible data sets:

\begin{itemize}
\item TODO
\end{itemize}

Forbidden Datesets: Some data sets should not be used in the project:

\begin{itemize}
\item Titanic (R data set)
\item mtcars (R data set)
\item baseball batting (the one used in Bob Carpenter's case study)
\end{itemize}

Modelling requirements and recommendations:
\begin{itemize}
\item Every parameter needs to have an explicit proper prior. Improper flat priors (default in Stan) are not allowed.
\item An hierarchical model is a model, where the prior of certain parameters contains other parameters, which are also estimated in the model. For example. \texttt{b $\sim$ normal(mu, sigma)} with \texttt{mu $\sim$ normal(0, 1)} and \texttt{sigma $\sim$ exponential(1)}
\item If a parameter does not have both a natural lower and upper boundary, \texttt{uniform(a, b)} are considered bad practice as they set hard boundaries on parameters without such natural hard boundaries.
\item Models should ideally use at least one covariate. That is, just modeling
your outcome variable without doing any predictions or similar using other variables, is likely too simple of an analysis for this project.
\item Using brms for the model specification is allowed, but the Stan code needs to be included, briefly commented, and all priors need to be checked from the Stan code. 
\end{itemize}


%\end{itemize}

\section*{Peergrade rubric}

Part of the questions are used to check that the minimal requirements
of the project work are included in the notebook. Most of the questions are for giving
feedback to other students. The received feedback and your response to
that will be discussed in the evaluation meeting.
%Peergrade score you receive is not you final grade for the project work.

\begin{itemize}
\item Can you open the notebook?
\begin{itemize}
\item yes
\item no
\end{itemize}

\item Is there an introduction? 
  \begin{itemize}
  \item There is no clear introduction
  \item The introduction touches on the main topic
  \item The introduction states the main topic and provides an overview of the notebook
  \item The introduction is inviting, presents an overview of the
    notebook. Information is relevant and presented in a logical
    order.
\end{itemize}

\item What are your suggestions on how to improve the introduction?

\item Is there a conclusion? 
  \begin{itemize}
  \item There is no clear conclusion
  \item A conclusion is included
  \item The conclusion is clear
  \end{itemize}
  
Describe in your own words what is the main conclusion of the data analysis in this notebook?

\item The structure and organization of the notebook
  \begin{itemize}
  \item The notebook lacks a clear data analysis story
  \item The notebook attempts to tell a coherent data analysis story but lacks some focus and clarity.
  \item The notebook presents a clear cohesive data analysis story
  \item The notebook presents a clear cohesive data analysis
    story, which is enjoyable to read
  \end{itemize}
  
\item Overall, what did you think of the structure and organization of the
notebook? Name at least one way your peer could improve structure and
organization.

\item Accuracy of use of statistical terms
  \begin{itemize}
  \item There are numerous errors in use statistical terms
  \item There are some errors in use of statistical terms
  \item Statistical terms are used accurately but sometimes lack clarity
  \item Statistical terms are used accurately and with clarity
  \end{itemize}
  
\item  Description of the data, and the analysis problem
  \begin{itemize}
  \item yes
  \item no
  \item Did you get a sense of what is the data and the analysis problem when they were first introduced? Where and how might the author make the model description more clear?
\end{itemize}

\item Are there more than one model
  \begin{itemize}
  \item yes
  \item no
  \item Was it easy to find the list of the models?
\end{itemize}

\item Description of the models
  \begin{itemize}
  \item yes
  \item no
  \item Did you get a sense of what the models are? Where and 
  how might the author make the model description more clear?
\end{itemize}

\item Description of the prior choices
  \begin{itemize}
  \item No priors or improper priors (e.g. uniform on unconstrained parameter) used 
  \item Priors listed but not justified
  \item Priors are listed and justified
  \end{itemize}
  
\item Is Stan code included?
  \begin{itemize}
  \item yes
  \item no
  \end{itemize}

  
\item Is the code for how Stan model is run included?
  \begin{itemize}
  \item yes
  \item no
  \end{itemize}

  \item Is Rhat convergence diagnostics included?
    \begin{itemize}
    \item No
    \item Yes, but no discussion what can be concluded from the shown Rhat values
    \item Yes, with discussion what can be concluded from the shown Rhat values
  \end{itemize}

  \item Are HMC specific convergence diagnostics (divergences, tree depth) included?
    \begin{itemize}
    \item No
    \item Yes, but no discussion what can be concluded from the shown values
    \item Yes, with discussion what can be concluded from the shown values
  \end{itemize}


  \item Is effective sample size diagnostic (ESS) included?
    \begin{itemize}
    \item No
    \item Yes, but no discussion what can be concluded from the shown values
    \item Yes, with discussion what can be concluded from the shown values
  \end{itemize}

\item Is there posterior predictive checking?
  \begin{itemize}
  \item No
  \item Yes, but no discussion what can be concluded from the shown checks
  \item Yes, with discussion what can be concluded from the shown checks
  \end{itemize}

\item Is there a discussion of problems and potential improvements ?
  \begin{itemize}
  \item No
  \item Some
  \item Very clear
  \end{itemize}

\item Choose something you like about the notebook and explain why you like it. 

\item If you were to go back and redo your own notebook after reading this submission, what would you change?

\item If the student(s) were to complete this project work again, what could they change, to make it overall better?
\end{itemize}





\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:
